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Abstract 

In  order  to  explore  the  potential  impact  of  novel  command  and  control  configurations, 
it  is  useful  to  have  some  means  of  extrapolating  from  existing  systems  and  comparing 
the  outcome  of  change  from  existing  to  novel  systems.  By  taking  a  ‘systems’  view  of 
operations,  it  is  possible  to  consider  the  impact  of  reconfiguration  upon  the 
performance  of  the  system  and  upon  the  agents  operating  within  the  system.  The  aim 
of  the  WESTT  analytical  prototyping  tool  is  to  support  systems  analysis  and  to  allow 
the  analyst  to  explore  the  impact  of  reconfiguration  through  the  manipulation  of 
models.  In  this  paper  we  describe  the  underlying  approach  behind  the  tool  which 
emphasises  taking  multiple  perspectives  (task/social  network/knowledge  network) 
with  regard  to  the  activity  being  analysed  and  we  describe  the  functionality  of  the 
present  version  of  the  software. 

Introduction . 

The  WESTT  (Workload,  Error,  Situational  Awareness,  Time  and  Teamwork) 
software  tool  is  currently  undergoing  iterative  development  by  the  Human  Factors 
Integration  Defence  Technology  Centre,  UK  (HFIDTC).  The  motivation  for  the 
development  of  the  tool  was  to  produce  a  means  of  visualising,  measuring  and 
modelling  C2  and  team  activity.  WESTT  represents  team  activity  at  a  systems  level  in 
which  both  humans  (and  the  technologies  they  interact  with)  are  represented  as 
agents.  Representing  humans  and  technology  at  an  equal  level  is  integral  to 
understanding  their  relationship  and  also  allows  one  to  directly  model  the  impact  of 
augmenting  or  even  replacing  team  members  with  new  technology.  In  addition  to  its 
role  in  processing  empirical  data,  WESTT  provides  a  means  of  evaluating  systems- 
level  descriptions  of  operations  with  a  view  to  supporting  analytical  prototyping. 
Analytical  prototyping  is  based  on  the  concept  that  initial  system  descriptions  can  be 
quantitatively  explored  in  order  to  evaluate  the  potential  benefits  of  modification 
(Baber  and  Stanton  1999,  2001).  In  other  words,  a  prototyping  approach  to  design  can 
be  taken  in  which  an  initial  system  specification  -  perhaps  based  on  observations  of 
current  operations  -  can  be  quantified  and  then  modified;  then  with  each  iteration  of 
the  process,  the  resulting  system  can  be  quantified  and  compared  with  the  initial 
description  in  order  to  evaluate  the  scale  of  likely  improvements.  In  summary 
WESTT  has  a  range  of  possible  applications: 

•  Analysis  of  field  data  and  evaluation  of  current  practice 

•  Comparison  between  actual  performance  and  the  design/doctrine 

•  Evaluation  of  changes  to  current  practice  through  modelling 

•  Evaluation  of  performance  in  training  and  virtual  environments 

It  is  important  to  note  that  WESTT  does  not  in  its  current  form  offer  the  means  to 
produce  full-blown  simulations  of  the  battlefield  but  rather  is  an  accessible  tool  for 


rapid  prototyping  and  assessment  in  the  hands  of  an  analyst  who  may  use  it  together 
with  their  expertise,  existing  simulation  tools  and  data.  The  analyses  offered  by 
WESTT  may  give  strong  indications  about  the  implications  of  certain  patterns  of 
activity  but  the  tool  does  not  in  itself  generate  statements  about  how  these  problems 
might  be  rectified  or  speak  directly  to  issues  of  operational  effectiveness.  Instead, 
WESTT  allows  the  analyst  to  quickly  evaluate  different  approaches  and  candidate 
solutions  with  a  view  to  the  predicted  risks  of  error,  time  delay,  workload, 
inappropriate  social  network  configuration  and  the  distribution  of  knowledge  in  the 
system  as  a  whole. 

Underlying  approach  and  overview. 

Key  to  the  design  of  WESTT  is  the  contention  that,  with  the  advent  of  network- 
enabled  systems  and  the  increased  recognition  of  the  importance  of  knowledge 
management  to  war  fighting,  fully  understanding  C2  performance  requires  an 
appreciation  of  what  happened  in  a  given  situation  (or  what  is  predicted  might 
happen)  from  at  least  three  distinct  but  closely  interrelated  perspectives:  the  tasks 
being  performed  by  the  team,  the  nature  of  the  social  network  the  team  is  acting 
within  and  the  knowledge  being  used  and  exchanged.  Thus  we  suggest  that  C2  and 
team  activity  is  best  understood  as  being  the  product  of  a  ‘network  of  networks’ 
wherein  the  examination  of  one  network  requires  knowledge  of  the  structure  of  the 
others  (all  working  within  a  technologically  constrained  network).  This  interrelation  is 
depicted  in  Figure  1.  Whilst  WESTT  is  ostensibly  a  software  tool,  it  should  also  be 
noted  it  implies  (to  some  extent)  a  particular  methodological  approach  to  the  study  of 
team  and  C2  behaviour. 


Figure  1 .  Relationship  and  mappings  between  task  network,  social  network  and 
propositional  knowledge  network  to  form  a  “network  of  networks”. 


An  implication  of  this  view  is  that  changing  an  element  of  one  network  has 
repercussions  within  the  other  two  networks.  For  example,  adjusting  manning  to  meet 
performance  targets  in  the  task  network  will  have  repercussions  for  both  the  social 
network  within  the  team  and  for  the  distribution  and  flow  of  knowledge  around  the 
team,  both  of  which  will  have  further  repercussions  upon  task  performance  itself. 
These  changes  may  of  course  be  either  beneficial  or  harmful  to  overall  performance. 
WESTT  then  allows  analysts  to  investigate  these  complexities  to  prevent  unforeseen 
damage  occurring  (e.g.,  creating  bottlenecks  in  information  flow)  and  also  to  help  an 
analyst  envisage  and  test  original  solutions  to  problems  (e.g.,  it  is  possible  that  what 
appears  to  be  a  manning  problem  may  actually  be  a  symptom  of  an  underlying 
knowledge  distribution  problem  that  can  be  solved  without  changing  the  manning 
itself). 

Our  approach  has  been  formed  with  primary  regard  to  military  operations,  but  we 
believe  the  WESTT  tool  can  be  used  widely  in  the  study  of  any  organisation  where 
C2  and  team  performance  are  important.  Indeed,  the  screenshots  in  the  present  paper 
are  of  emergency  service  (Police  and  Fire)  operations  we  have  already  analysed  using 
the  tool  (e.g.,  Stanton  et  al.,  2005). 


CASA. 

The  C4I  Activity  Sampling  Application  (CASA)  is  a  companion  application  to 
WESTT  designed  to  support  the  collection  of  field  observation  data.  The  software 
currently  runs  on  Personal  Digital  Assistants  (PDAs)  like  the  iPAQ  and  on  portable 
tablet  computers  for  easy  deployment  in  the  field.  As  we  will  see  shortly,  WESTT  by 
default  represents  team  activity  using  a  comprehensive  taxonomy  of  actions  the  same 
as  those  found  in  Operation  Sequence  Diagrams;  these  include  Transmit,  Receive, 
Inspect,  Operate,  Decide  and  Store.  On  observing  a  given  action  the  analyst  need  only 
click  on  the  screen  to  identify  the  actor  and  click  to  identify  the  action.  These  data  are 
then  time  stamped  and  saved.  If  the  analyst  selects  Transmit  or  Receive,  a  further 
screen  is  displayed  to  allow  selection  of  the  medium  being  used,  i.e.,  Visual, 
Electrical,  Sound  /  Verbal,  Internal  Communication,  External  Communication,  Touch, 
Mechanical,  Walking,  and  Hand  delivered.  After  defining  the  medium  of 
communication  the  analyst  then  specifies  the  agents  sending  and  receiving  the 
communication.  On  completing  the  observation,  the  analyst  then  is  taken  to  an 
‘overview’  screen.  This  shows  the  details  of  the  observation  (entered  on  the  initial 
screen)  together  with  a  scrolling  window  showing  previous  observations.  The 
presentation  of  the  observations  in  this  format  allows  initial  review  of  the  data.  Upon 
returning  from  the  field,  the  analyst  can  simply  transfer  the  comma- separated  data 
directly  into  WESTT  or  else  combine/edit  observations  in  a  spreadsheet  for 
subsequent  export  to  WESTT.  Three  screens  from  the  CASA  user  interface  are 
depicted  in  Figure  2. 
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Figure  2.  CASA  interface  screens 


Whilst  CASA  has  been  produced  to  reduce  the  difficulty  of  field  observations,  it 
should  also  be  noted  that  one  might  instead  choose  to  use  pen  and  paper  in  the  field 
and  simply  transcribe  the  results  into  a  suitable  spreadsheet  back  at  base.  One  might 
also  use  official  documents  to  produce  a  representation  of  “model”  performance.  We 
have  also  had  success  utilising  official  event  logs  in  studies  of  Police  operations  and 
envisage  it  might  also  be  possible  to  use  data  feeds  from  simulators,  virtual 
environments  and  network-enabled  systems  themselves. 


Main  data  table. 


At  the  heart  of  WESTT,  in  terms  of  both  the  user  interface  and  the  underlying  method 
of  analysis  is  the  main  data  table  (see  Figure  2  which  depicts  the  architecture  of  the 
system).  The  data  table,  as  the  name  implies,  simply  displays  an  ordered  list  of  events 
over  time  together  with  the  actors  involved  and  other  relevant  details  (labels  breaking 
the  action  down  by  phase  and  other  descriptive  material  can  be  added  for  example).  In 
essence  the  data  table  is  very  similar  to  a  spreadsheet  and  has  similar  editing 
capabilities;  in  addition  one  can  also  ask  WESTT  to  check  that  all  the  observations  in 
the  table  fall  in  chronological  order  to  prevent  errors  in  analysis.  A  screenshot  of  the 
main  data  table  is  shown  in  Figure  3. 

It  is  through  the  manipulation  of  this  data-table  that  the  analyst  can  try  out  different 
approaches.  For  example,  a  team  of  Firemen  may  have  been  recorded  putting  out  a 
blaze.  The  analyst  may  wish  to  investigate  how  adding  an  extra  level  of  command 
might  have  affected  events  in  response  to  workload  problems  reported  by  Fire 
commanders  (e.g.,  adding  an  extra  agent  into  the  scenario  and  transferring  some 
command  activities  and  decisions  to  them).  It  may  be  the  case  that  this  solution  works 
and  reduces  the  workload  but  alternatively  modelling  of  the  time  taken  to  pass 
communications  and  make  decisions  along  the  chain  of  command  may  reveal  this  is 
inadvisable  if  the  observed  timeframe  for  events  is  to  be  met.  Alternatively  one  might 
want  to  remove  a  Fireman  from  the  team  and  see  if  the  remaining  work  can  be  split  up 
amongst  those  that  remain  without  any  individual  being  overburdened.  It  might 
quickly  be  observed  somebody  needs  to  be  in  two  places  at  one  time  or  perhaps  it 
would  require  one  Fireman  to  be  skilled  in  two  very  different  areas  of  expertise.  He 
may  even  be  called  upon  to  have  knowledge  of  ongoing  events  he  has  no  way  of 
knowing  about.  Such  things  may  be  far  from  obvious  in  a  complex  scenario  and 
WESTT  is  an  attempt  to  aid  analysts  in  dealing  with  the  high  levels  of  complexity 


distributed  team  performance  in  NEC-enabled  contexts  can  produce.  Rather  than 
computing  optimal  solutions  to  problems  itself,  the  tool  encourages  a  different  sort  of 
process,  namely  analytical  prototyping,  wherein  the  analyst  can  rapidly  try  out  of 
‘what  if  scenarios  and  “tweak”  the  design  with  feedback  in  the  form  of  ongoing 
automatic  analyses  of  the  implications  of  those  changes. 


Figure  3.  The  main  data  table. 

Operation  sequence  diagram  /  UML. 

On  the  basis  of  the  information  in  the  main  data  table,  WESTT  will  automatically 
draw  and  label  an  Operation  Sequence  Diagram  (OSD).  The  use  of  the  OSD  has  been 
widespread  within  the  Ergonomics  community  since  the  1950s  and  provides  a  means 
of  representing  the  activity  of  agents  within  a  system  over  the  course  of  a  mission.  As 
Meister  (1985)  pointed  out,  “The  OSD  can  be  drawn  at  a  system  or  task  level  and  it 
can  be  utilized  at  any  time  in  the  system  development  cycle  provided  the  necessary 
information  is  available.  It  can  aid  the  analyst  in  examining  the  behavioural 
implementation  of  design  alternatives  by  permitting  the  comparison  of  actions 
involved  in  these  design  alternatives.”  [p.  67].  However,  the  use  of  OSDs  has  hitherto 
been  beset  by  problems  in  their  creation  “. .  .the  task  of  drawing  a  complex  OSD  can 
be  extremely  cumbersome  and  expensive.”  (Meister,  1985,  p.  68).  Various  attempts 
have  been  made  to  automate  the  drawing  of  OSDs  (particularly  during  the  1980s), 
although  there  are  few  if  any  commercial  products  that  are  currently  available  to  do 
this.  WESTT  therefore  fills  this  gap  (Figure  4). 
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Figure  4.  Operation  Sequence  Diagram 

In  addition  to  the  OSD,  WESTT  can  also  automatically  generate  Unified  Modelling 
Language  (UML)  diagrams,  namely,  a  Sequence  diagram  and  a  Use  Case  diagram. 
Whilst  the  OSD  may  be  familiar  to  Human  Factors  and  Operations  Research  experts, 
in  Engineering  (and  particularly  Software  Engineering)  the  UML  system  is  both 
popular  and  widely  understood.  Thus  to  make  the  tool  useful  to  as  wide  an  audience 
as  possible  and  to  facilitate  communication  between  Human  Factors  analysts  and 
Engineers  involved  in  the  production  of  C2  technology  we  decided  to  add  these  UML 
options.  The  Sequence  diagram  (see  Figure  5a)  is  very  similar  to  OSD  in  so  far  as  it 
portrays  much  the  same  information  (actions  by  agent  over  time)  albeit  using  a 
slightly  different  layout  and  symbology.  The  Use  Case  diagram  displays  the 
associations  between  individual  actors  and  the  tasks  (termed  Use  Cases  herein)  that 
they  are  involved  in  (see  Figure  5b).  By  offering  a  range  of  representations  of  the  data 
we  allow  the  analyst  to  pick  the  most  relevant  to  their  needs  at  the  time.  For  example, 
if  one  is  interested  in  which  agents  are  collaborating  on  particular  tasks  but  have  no 
interest  in  the  timings  involved,  the  Use  Case  diagram  may  be  a  more  preferable 
rendering  of  the  information  then  the  OSD  or  the  UML  Sequence  diagram. 
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Figure  5.  (a)  UML  use  case  diagram  (b)  UML  Sequence  diagram  respectively 


Social  Network. 

Teamwork  is  explored  through  concepts  from  Social  Network  Analysis.  On  the  basis 
of  the  central  data  table  (i.e.,  the  same  data  used  to  generate  the  OSD  and  UML 
views)  WESTT  also  automatically  draws  a  Social  Network  diagram  that  graphically 
portrays  the  interconnections  between  agents  within  a  system.  Each  agent  is 
represented  as  a  node  and  is  connected  to  others  via  lines  termed  “edges”  (see  Figure 
6).  Again,  whilst  there  are  products  that  will  draw  social  networks,  WESTT  is  notable 
for  allowing  one  to  go  directly  from  empirical  data  to  a  full  social  network  in  one  step 
together  with  the  OSD  as  a  representation  of  the  context  from  which  the  social 
network  arises  (it  is  axiomatic  within  the  social  network  analysis  community  that 
interpretations  of  social  networks  are  only  valid  with  careful  reference  to  the 
situations  that  produce  them).  Qualitative  analysis  of  social  networks  can  yield 
interesting  results  in  and  of  itself;  one  can  for  example  identify  nodes  that  are  acting 
as  hubs  that  connect  other  nodes.  In  some  cases  this  function  may  not  have  been 
deliberately  assigned  to  an  individual  with  implications  for  their  performance  in  other 
tasks.  Whilst  we  often  envisage  activity  in  teams  as  occurring  with  regard  to  specific 
hierarchies  and  protocols,  upon  observation  it  can  often  be  seen  simply  from  looking 
at  a  social  network  diagram  that  the  reality  is  somewhat  different. 

Social  network  analysis  is  based  on  the  simple  intuition  that  the  structure  of 
relationships  between  actors  plays  a  determinant  role  in  the  performance  or  action  of 
that  social  network  and  the  actors  within  it.  It  is  worth  noting  from  the  outset  that 
social  network  theory  is  based  upon  empiricism  and  mathematics  (indeed,  modem 
social  network  analysis  techniques  would  not  exist  had  Graph  Theory  not  undergone 
rapid  development  as  a  mathematical  field  in  the  1970s).  Today,  social  network 


theory  is  widely  used  across  myriad  disciplines;  it  can  be  used  as  a  tool  to  investigate 
organisations,  decision  making,  the  spread  of  information,  the  spread  of  disease, 
mental  health  support  systems,  anthropology,  child  development  etc.  Most  recently 
there  has  been  a  great  deal  of  enthusiasm  for  the  using  the  techniques  of  social 
network  analysis  (SNA  hereon)  to  study  the  internet  and  connections  between  both 
web  pages  (e.g.,  Google  ‘page  rank’  technology)  and  internet  users  (e.g.,  see  Adamic, 
Buyukkoten  &  Adar,  2003).  In  terms  of  studying  the  architectures  encountered  in  the 
increasingly  complex  Network  Enabled  command  and  control  networks  (both  pre¬ 
designed  and  formed  ad  hoc )  SNA  appears  therefore  to  be  the  logical  choice  of 
analysis  tool.  Modelling  of  military  command  and  control  networks  using  this  general 
approach  has  already  yielded  intriguing  results  (e.g.,  Dekker,  2002). 
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Figure  6.  Social  Network  diagram  and  Social  network  matrix 

As  well  as  giving  the  analyst  a  visual  representation  of  the  relationships  between 
agents  which  can  be  used  to  understand  the  general  character  of  the  social  network, 
the  data  on  which  it  is  based  can  be  analysed  using  algorithms  and  statistical 
techniques  (for  a  comprehensive  review  see  Wasserman  &  Faust,  1992).  For  our 
purposes  these  metrics  fall  into  two  main  camps;  measures  of  the  activity  of  nodes 
and  measures  that  pertain  to  the  topography  of  the  network.  Each  class  of  metric 
provides  a  different  perspective  on  the  structure  and  performance  of  the  social 
network.  A  measure  of  node  activity  implemented  in  WESTT  is  Sociometric  status, 
which  gives  an  indication  of  the  contribution  a  given  node  makes  to  the  overall 
amount  of  communication  in  the  network. 


The  other  two  social  network  metrics  currently  instantiated  in  WESTT  are  geodesic 
distance  and  centrality  which  relate  more  directly  to  the  form  of  the  network. 

Geodesic  distance  refers  to  the  shortest  possible  path  between  two  nodes  in  a  network 
and  thus  can  be  assumed  to  be  shortest  path  for  a  communication  to  pass  between  two 
agents.  Typically,  the  greater  the  geodesic  distance  between  two  agents,  the  longer 
information  will  take  to  propagate  from  one  to  another  and  the  greater  the  risk  that 
information  will  lose  its  value  both  because  of  the  degradation  encountered  in 
inaccurate  reception  or  retransmission  (as  in  a  game  of  ‘Chinese  whispers’)  and  in 
terms  of  the  information  pertaining  to  a  rapidly  changing  situation  being  rendered 
inaccurate  before  it  reaches  its  eventual  recipient.  If  the  information  in  question  is 
intelligence  this  might  mean  plans  are  formed  or  orders  issued  that  are  inappropriate 
to  the  extant  situation  with  the  result  that  clumsy,  uncoordinated  or  even  hazardous 
actions  are  taken.  Centrality  is  an  overall  indication  of  how  close  a  node  is  in  terms  of 
geodesic  distance  from  the  all  the  other  nodes  in  the  network. 

Knowledge  objects  and  the  propositional  network . 

It  is  the  network  of  propositions  that  embodies  a  novel  conceptualisation  of  situation 
awareness  at  a  distributed  level  used  by  WESTT.  The  notion  of  shared  awareness, 
understanding  and  knowledge  is  an  important  part  of  the  NEC/NCW  vision  as  it 
contributes  critically  to  the  eventual  aim  of  units  being  coordinated  and  synchronised 
at  a  high  tempo  on  the  battlefield  (e.g.,  Alberts,  Garstka,  &  Stein,  2003;  MoD,  2004). 
Given  the  importance  of  information  sharing  and  knowledge  management  to  the 
contemporary  analysis  of  C2  and  the  fact  that  our  approach  is  somewhat  unusual,  we 
shall  spend  some  time  discussing  in  depth  how  it  works  and  the  process  by  which  we 
arrived  at  our  view.  The  question  of  how  to  usefully  represent  systemic 
understanding,  awareness  and  coordination  is  a  complex  one  and  no  doubt  there  are 
many  possibilities  that  can  be  investigated  with  relevance  to  different  requirements 
and  levels  of  analysis.  Our  chosen  approach  for  WESTT  has  been  to  use  propositional 
networks  of  knowledge  in  a  manner  inspired  by  theories  of  distributed  cognition. 

Distributed  cognition  is  an  umbrella  term  used  to  describe  a  perspective  that  views 
system  performance  as  a  cognitive  activity  shared  across  individuals,  their 
technology,  artifacts  and  the  environments  in  which  all  these  things  are  found.  Whilst 
it  is  perhaps  most  common  to  view  cognitive  activities  as  things  that  occur  in  the 
minds  of  individuals  (or  perhaps  within  machines  that  embody  some  degree  of 
artificial  intelligence),  proponents  of  distributed  cognition  claim  that  cognitive 
activities  (that  is,  the  creation,  transmission,  processing,  representation  and  re¬ 
representation  of  information  to  meet  a  goal)  can  all  can  identified  within  distributed 
systems  that  are  made  up  of  multiple  actors  and  can  in  some  cases  be  directly 
observed  by  a  third  party  (see  Hutchins,  1995a). 

A  cognitive  system  moves  towards  its  goal-state  by  performing  transformations  upon 
representations  and  this  is  true  of  individual  cognition  and  distributed  cognition 
(Perry,  2003).  Within  a  distributed  cognitive  system,  transformations  of 
representations  are  achieved  by  the  combination,  interpretation  and  re-presentation  of 
information  provided  by  the  artifacts  and  individuals  in  the  system  (Hutchins,  1995a; 
Artman  and  Garbis,  1998).  Artifacts  are  objects  (and  for  want  of  a  better  word, 
“things”)  in  the  environment  that  permit  a  cognitive  activity  (e.g.,  pen  and  paper 
constitute  a  very  common  artifact  that  can  transcribe  and  then  hold  information  in  an 


accessible  form  that  can  be  accessed  by  another  actor).  Furthermore,  distributed 
cognition  assumes  that,  in  some  circumstances,  actions  performed  by  individuals  can 
lead  to  information  processing  directly  at  the  systems  level ,  that  is,  without  first 
requiring  individual-level  information  processing.  A  trivial  example  of  this  would  be 
that  one  can  deliver  a  letter,  thus  transmitting  and  sharing  information,  without 
actually  reading  the  letter  itself.  Alternatively  a  person  might  walk  through  a  muddy 
field  leaving  a  set  of  footprints.  The  individual  might  not  after  the  event  have  a 
detailed  recollection  of  their  course  or  perhaps  may  have  been  wandering  aimlessly  in 
a  daydream,  but  the  information  is  there  to  be  found  in  the  environment.  In  short,  the 
very  act  of  manipulating  an  object  in,  or  interacting  with,  the  world  can  be  sufficient 
to  lead  to  the  consequences  of  information-processing  with  or  without  an  actor 
directly  contemplating  or  processing  the  information. 

A  classic  exposition  of  the  principles  of  distributed  cognition  has  been  given  in  a 
description  of  the  way  in  which  an  aircraft  cockpit  ‘remembers’  the  speed  of  the 
aircraft  (Hutchins,  1995b).  The  paper  provides  an  account  of  the  system  interactions 
that  occur  in  order  to  answer  the  apparently  simple  question  of  how  fast  an  aircraft  is 
flying.  In  the  paper,  Hutchins  (1995b)  assumes  that  systems  have  goals;  in  an  aircraft 
system,  the  goal  is  to  successfully  complete  the  flight  and  a  subgoal  within  this  is 
achieving  an  appropriate  flight  speed.  Components  within  this  system  might  share  a 
goal,  and  can  perform  actions  to  achieve  sub  goals  in  pursuit  of  the  overall  goal,  but 
there  is  no  one  component  that  can  directly  claim  to  have  completely  met  the  goal  in 
isolation,  it  is  a  combination  of  the  different  parts  of  the  aircraft  and  the  pilot.  It  is 
important  to  note  then  that  studying  a  distributed  system  from  a  distributed  cognition 
perspective  is  not  about  summating  studies  of  individual  knowledge  or  awareness; 
rather  it  is  about  describing  the  system  as  a  whole.  The  properties  a  system  might 
have  overall  may  not  necessarily  be  found  through  individual  examination  of  its 
constituent  parts;  in  other  words  distributed  cognition  is  an  emergent  property  of  a 
distributed  cognitive  system.  In  prospect  then,  just  as  individual  behavior  can  be 
understood  with  reference  to  theories  of  human  cognition,  so  the  behavior  of 
distributed  systems  can  be  understood  with  reference  to  notions  of  distributed 
cognition,  with,  at  face  value  at  least,  obvious  appeal  for  those  interested  in 
understanding  the  way  in  which  Network-Enabled  socio-technical  distributed  systems 
may  behave. 

One  difficulty  in  utilizing  the  theory  of  distributed  cognition  in  a  practical  way  is  that 
its  authors  have  generally  taken  an  ethnographic/reportage  approach,  writing 
descriptions  of  activities  they  have  observed  and  discussing  how  these  might  be 
interpreted  as  cognition.  This  would  clearly  be  of  little  use  within  a  rapid  prototyping 
tool  such  as  WESTT,  thus  we  have  attempted  to  operationalise  some  useful  aspects  of 
the  distributed  cognition  perspective.  For  the  purposes  of  the  present  work  then,  a  key 
challenge  was  to  find  a  way  of  illustrating  what  information  is  being  represented 
within  the  system,  and  to  show  which  agents  have  access  to  different  aspects  of  the 
represented  information  at  different  times.  In  other  words,  our  purpose  is  to  identify 
solely  the  information  used;  we  leave  the  question  of  how  to  describe  changing 
representations  within  the  system  to  later  work.  Following  the  assumption  that  a 
distributed  cognition  system  embodies  the  same  cognitive  processes  as  an  individual 
might  (albeit  spread  across  a  system),  we  suggest  that  a  means  of  describing 
knowledge  in  individuals,  namely  propositional  networks,  can  be  profitably  applied  to 
the  representation  of  the  action  of  and  within  a  larger  socio-technical  system. 


The  basic  approach  is  for  the  analyst  to  collect  information  from  debrief,  interviews, 
procedural  manuals  and  other  sources,  to  define  the  ‘knowledge  objects’  relevant  to 
the  mission.  These  knowledge  objects  are  then  connected  by  linkages  based  on 
semantic  propositions  that  define  their  relationship  (e.g.,  IS,  HAS,  KNOWS),  thus  a 
network  of  knowledge  is  produced  (see  for  example,  Anderson,  1983).  In  terms  of 
constraining  what  a  proposition  can  be  we  take  Anderson’s  approach  that  a 
proposition  is  “. .  .the  smallest  unit  about  which  it  makes  sense  to  make  the  judgment 
true  or  false”  (Anderson,  1980,  p.102).  To-date  we  have  found  the  best  way  to  elicit 
knowledge  objects  has  been  to  use  the  Critical  Decision  method  for  structuring 
interviews.  The  Critical  Decision  method  (Klein,  1989)  is  a  form  of  critical  incident 
technique.  According  to  Klein  (1989),  “The  CDM  is  a  retrospective  interview 
strategy  that  applies  a  set  of  cognitive  probes  to  actual  non-routine  incidents  that 
required  expert  judgment  or  decision  making”  (p.  464).  In  our  implementation  of  this 
approach  the  interview  proceeds  through  a  series  of  four  stages:  briefing  and  initial 
recall  of  incidents;  identifying  decision  points  in  specific  incidents;  probing  the 
decision  points;  and  finally  checking.  The  knowledge  objects  themselves  take  myriad 
forms;  because  they  define  the  tasks  and  the  knowledge  needed  to  successfully 
complete  them  they  include  information  about  the  scenario,  pre-existing  knowledge 
and  skills  actors  must  have  (SOPs,  cultural  knowledge,  doctrinal  knowledge  etc)  and 
information  that  must  be  found  in  the  scenario  or  environment  to  successfully 
complete  a  task.  In  many  cases  this  would  also  likely  include  knowing  about  what 
other  members  of  a  team  are  doing  although  not  necessarily  the  sharing  of 
information  directly  between  actors. 

In  terms  of  WESTT  itself,  the  knowledge  objects  are  entered  into  a  matrix  which 
allows  the  analyst  to  define  the  relationship  between  the  objects  (see  Figure  7). 
WESTT  is  then  able  to  automatically  provide  a  graphical  representation  of  the 
‘knowledge  space’  involved  in  the  scenario  (Figure  8).  Typically,  this  network  is  then 
presented  to  Subject  Matter  Experts  (SME)  in  order  to  validate  the  level  of  detail  and 
the  inclusion  of  specific  knowledge  objects.  Once  the  network  has  reached  an 
acceptable  state,  it  can  be  subjected  to  network  analysis  in  a  similar  way  to  social 
networks  to  identify  trends  in  the  relationships  between  knowledge  objects.  Nodes  in 
a  PN  can  be  associated  with  specific  agents  through  a  system  of  colour  coding  (as  can 
be  seen  in  Figure  8)  and  also  WESTT  provides  a  facility  to  “play  back”  the  spread  of 
activation  in  the  propositional  network  over  time  so  the  analyst  can  watch  the  spread 
of  knowledge  use  and  sharing  over  the  duration  of  the  scenario. 


Figure  7.  Knowledge  object  matrix  with  propositions 
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Figure  8.  Propositional  network  of  knowledge  items  and  scenario  playback  controls. 


From  a  theoretically  agnostic  perspective,  the  propositional  network  approach  to  the 
analysis  of  tasks  and  incidents  produces  data  regarding  which  information  is  most 
important  (and  to  whom)  at  different  points  over  time  and  also  serves  to  give  a  fuller 
description  of  the  tasks  to  which  it  is  linked,  so  it  is  not  necessarily  the  case  that  our 
explanation  of  what  this  means  be  accepted  fully  for  this  to  be  a  useful  feature  of 
WESTT  as  a  software  tool.  However,  from  a  theoretical  perspective  we  are  prompted 
to  ask  what  it  is  in  semantic  terms  that  we  are  representing.  If  we  argue  that  we  are 
identifying  who  at  different  points  in  time  makes  use  of  different  knowledge  objects 
and  that  these  are  the  focal  points  that  the  distributed  cognition  system  is 
concentrating  upon,  then  we  can  argue  that  the  dynamic  pattern  of  activation  within 
the  propositional  network  represents  something  akin  to  a  system  level  depiction  of 
situational  awareness.  To  unpack  that  slightly;  the  ‘situation’  is  modeled  in  terms  of 
the  knowledge  objects  in  the  propositional  network  itself  and  the  ‘awareness’  (which 
changes  over  time  and  differs  between  actors)  is  the  pattern  of  activation  within  the 
network.  It  is  a  point  worth  referring  back  to  in  our  earlier  discussion  that  the  ‘mental 
state’  of  a  distributed  cognitive  system  is  an  emergent  property  of  that  system  and  is 
not  necessarily  held,  mirrored  in  or  apprehended  directly  by  any  one  individual.  In 
some  instances,  a  knowledge  object  will  be  activated  by  more  than  one  agent.  This 
would  indicate  the  potential  for  sharing  that  knowledge  object,  but  this  need  not  imply 
that  ‘sharing’  will  result  in  communication  or  shared  understanding.  For  example, 
different  agents  might  activate  a  knowledge  object  from  different  perspectives  or  for 
different  reasons,  which  could  result  in  different  interpretations  or  uses  of  that 
knowledge  object.  This  suggests  that  the  concept  of  ‘shared  awareness’  is  richer  than 
simply  assuming  mutual  knowledge  or  shared  understanding.  Rather,  a  knowledge 
object  can  be  jointly  activated,  can  be  interpreted  in  different  ways  and  can  be 
communicated  (or  not)  between  agents.  At  the  present  time  WESTT  offers  no  clear 
guidance  on  this  issue  (and  it  is  not  clear  to  us  at  the  present  time  if  indeed  it  should) 
but  in  practical  terms  an  analyst  can  get  a  qualitative  feel  for  the  context  of  activation 
by  double  clicking  on  an  activated  knowledge  object  which  is  hyperlinked  back  to  the 
main  data  table  screen  with  the  appropriate  phase  of  the  operation  clearly  marked.  Just 
as  a  social  network  cannot  be  fully  understood  without  careful  reference  to  the 
situation  that  produced  it  (and,  we  hold,  vice  versa),  we  feel  that  the  same  is  true  of 
our  propositional  network  of  knowledge. 

Workload,  Error  and  Time  metrics. 

The  final  part  of  WESTT  is  the  calculation  of  workload,  error  and  time  metrics.  All 
three  metrics  may  constrain,  from  a  Human  Factors  perspective,  the  design  of  a 
mission  or  set  of  procedures.  A  high  combined  error  probability  score  for  a  procedure 
may  render  it  too  risky  and  too  high  a  workload  score  would  indicate  the  placement  of 
sizable  burdens  upon  human  (or  even  technological)  agents.  Time  as  a  metric  differs 
slightly  in  that  it  indicates,  in  theory,  the  extent  to  which  a  set  of  tasks  might  under  or 
overrun.  Clearly  one  might  want  to  take  a  careful  look  at  events  or  procedures  where  a 
delay  would  be  a  critical  problem.  Combining  measures  of  error  and  time  can  create  a 
probabilistic  ‘worst  case’  analysis  showing  how  a  goal-oriented  task  might  overrun  in 
a  percentage  of  cases  after  an  error  occurs  and  must  be  recovered  from.  Whilst 
WESTT  does  not  directly  make  predictions  about  operational  effectiveness  per  se , 
clearly  time  scale  and  the  risk  of  human/technical  error  are  factors  worth  considering 
when  contemplating  the  chance  of  success  a  mission  or  activity  has.  In  terms  of  the 


software  itself,  this  process  of  metric  generation  is  done  by  combining  the  data  about 
what  tasks  are  performed  by  who  (as  found  in  the  main  data-table  and  as  depicted  in 
the  OSD  and  Use  Case  figures)  with  databases  of  the  time  tasks  typically  take  to 
perform  and  the  probability  of  error.  The  first  advantage  of  using  this  “plug-in” 
database  approach  is  that  one  can  use  existing  data  that  have  already  been  extensively 
validated  and  which  make  WESTT  on  some  level  compatible  with  other  forms  of 
analysis  which  use  the  same  databases.  The  second  advantage  of  this  approach  is  that 
one  can  also  produce  and  modify  databases  not  just  for  human  agents  but  also 
technological  agents  (e.g.,  an  automatic  targeting  system)  that  define  their  different 
estimated  error  rates  and  task  completion  times.  In  general  by  making  the  database 
system  extensible  individual  users  can  to  some  extent  tailor  WESTT  to  their  particular 
specialist  needs. 

At  present,  for  human  agents,  we  are  using  The  University  of  Birmingham’s  CORE¬ 
DATA  (Kirwan,  Basra  &  Taylor-Adams,  1997)  to  provide  Human  Error  Probabilities 
(HEP)  for  specific  tasks.  These  HEPs  can  be  used  to  populate  fault  trees  in  order  to 
calculate  approximate  error  rates  for  specific  combinations  of  tasks.  In  WESTT,  an 
Operation  can  be  broken  down  further  into  a  set  of  discrete  tasks,  and  the  HEP  of  each 
task  used  to  calculate  overall  error  rate  (e.g.,  a  transmit  operation  on  the  main  data- 
table  might  break  down  into  entering  a  sequence  of  digits  into  a  mobile  phone, 
speaking  and  then  pressing  the  end  call  button). 

Human  Factors  has  developed  a  range  of  measures  to  describe  how  busy  a  person  is  in 
terms  of  how  much  cognitive  and  physical  activity  they  are  required  to  perform.  In 
terms  of  predictive  analysis  of  workload,  the  general  approach  would  appear  to  follow 
the  notion  that  changes  in  activity  can  be  mapped  over  time  to  provide  an  index  of 
loading  (Parks  &  Boucek,  1989);  this  could  be  considered  as  a  function  of  task 
scheduling  (Moray,  Dessousky,  Kijowski  &  Adapathya,  1991)  or  in  terms  of 
competition  between  cognitive  resources  (North  &  Riley,  1989).  At  present  WESTT 
provides  a  simple  metric  for  workload  based  on  the  operations  performed  by  a  given 
individual  agent  during  a  defined  phase  of  the  mission.  This  is  derived  from  the  OSD 
and  provides  an  index  of  ‘operations  demand’.  However,  workload  is  also  a  cognitive 
function  and  subsequent  developments  of  the  workload  algorithm  is  WESTT  will  take 
into  account  the  number  of  knowledge  objects  (as  represented  in  the  Propositional 
Network)  to  provide  a  scaling  factor  for  the  operations  demand.  Presumably  a  task 
that  requires  many  knowledge  objects  is  one  that  requires  the  individual(s)  performing 
it  to  know  many  things.  As  such  we  might  therefore  expect  it  to  be  a  cognitively 
complex  task.  It  is  conceivable  that  this  may  not  always  follow  and  so  we  are 
investigating  at  present  the  practicalities  that  surround  this  idea. 

Given  that  each  operation  is  made  up  of  specific  elements,  the  assignment  of  time  to 
tasks  in  simply  a  matter  of  looking  values  up  in  the  database.  WESTT  can  combine 
these  times  into  a  simple  linear  model  of  individual  time-on-task,  i.e.,  by  summing  all 
the  times  in  manner  that  is  similar  to  keystroke  level  models  which  might  also  be 
taken  as  an  indicator  of  workload  (e.g.,  Card,  Moran  &  Newell,  1983).  As  an 
alternative  we  are  currently  implementing  a  method  of  extending  this  analysis  into  a 
critical  path  model  (e.g.,  Gray,  John  &  Atwood,  1993;  Baber  &  Mellor,  2001)  which 
can  better  account  for  the  parallelism  of  human  activity.  Essentially  the  critical  path 
technique  is  about  identifying  which  activities  may  safely  occur  in  parallel,  which 


tasks  are  dependent  on  the  completion  of  other  tasks  before  they  commence  and  thus 
ultimately  which  tasks  are  critical  to  maintaining  a  schedule  and  which  are  not. 


Discussion  and  conclusions. 

This  paper  has  described  the  design  and  use  of  the  WESTT  software  tool  together 
with  an  extended  account  of  perhaps  its  most  controversial  feature,  a  propositional 
network  representation  of  system  situation  awareness.  WESTT  is  a  desktop  tool  that 
allows  an  analyst  to  investigate  and  measure  aspects  of  team  performance  beginning 
with  a  table  of  events  which  may  be  freely  modified.  WESTT  combines  three  types  of 
network;  a  network  of  tasks,  a  social  network  of  agents  and  knowledge  network.  In 
addition  to  this  WESTT  also  brings  to  bear  existing  forms  of  analysis  of  workload, 
error  and  task  duration.  Because  these  analyses  and  metrics  are  largely  automated 
WESTT  can  be  used  as  an  analytical  prototyping  tool  allowing  one  to  rapidly 
compare  different  approaches  by  manipulating  data  in  the  main  data-table.  By 
supporting  analysis  at  several  levels,  it  is  possible  to  explore  the  effects  of  changes  in 
system  structure,  the  introduction  or  removal  of  knowledge  objects  (which  might  be 
operating  procedures,  cultural  knowledge  or  tactical  information)  and  the  replacement 
or  augmentation  of  human  agents  with  technology. 

As  an  integrative  tool,  many  individual  aspects  of  WESTT  have  already  been  subject 
to  extensive  testing,  academic  discussion  and  validation.  For  example,  there  is  a  large 
literature  discussing  the  uses  and  abuses  of  social  network  modelling  and  much  of  the 
workload,  error  and  time  metrics  are  in  common  use  by  Human  Factors  practitioners. 
The  use  of  “plug  in”  databases  of  task  time  and  error  also  allow  one  to  employ  pre¬ 
validated  and  reliability-tested  estimates.  Validation  of  WESTT  as  a  whole  has  so  far 
consisted  of  discussions  with  Subject  Matter  Experts  regarding  the  face  validity  of  the 
outputs  and  usefulness  of  comparing  different  types  of  representations  as  an  aid  to  the 
investigation  by  analysts  of  complex  datasets  (this  has  taken  place  with  regard  to 
Police,  Fire  and  power  industry  scenarios).  In  addition  to  this  we  are  currently 
following  two  more  formal  approaches  to  validation.  First,  we  are  comparing  WESTT 
predictions  with  outcomes  delivered  by  simulation  tools  (such  as  MicroS  AINT). 
Second,  and  perhaps  more  importantly  given  an  analysis  tool’s  ultimate  duty  is  to 
produce  metrics  and  data  that  accord  with  reality  itself,  we  are  involved  in  ongoing 
experimental  work  looking  at  the  effects  of  different  network  structures  and 
technologies  upon  the  activities  of  teams  of  human  participants  in  different  games  and 
scenarios  and  comparing  the  outputs  of  WESTT  analyses  against  those  real  world 
patterns  of  performance. 

Future  directions  for  the  development  of  WESTT  include  adding  more  metrics  as 
appropriate  and  in  particular  “headline  figures”  which  more  succinctly  describe  the 
performance  of  the  “network  of  networks”  as  a  whole.  A  further  aim  is  to  make  more 
explicit  the  technical  constraints  that  exist  in  designing  NEC  socio-technical  systems. 
Networking  technology  itself  is  not  as  such  explicitly  described  and  would,  for  the 
time  being,  have  to  be  included  as  a  set  of  agents  that  received  a  communication  from 
one  agent  and  then  relayed  it,  possibly  via  other  intermediary  agents  that  are  part  of 
the  communication  system,  to  another.  These  activities  (transmit,  receive,  and  store 
information)  would  have  durations  and  error  probabilities  described  in  the  appropriate 


task  database.  This  might  be  an  appropriate  way  to  study  such  developments  in  some 
cases  (particularly  where  the  technology  might  be  said  to  embody  some  degree  of 
intelligence  and  internal  complexity),  but  could  become  difficult  or  cumbersome  to 
implement  in  large,  complex  scenarios.  A  possible  way  of  including  these 
communications  constraints  more  easily  within  our  construct  is  to  allow  the  definition 
of  a  communications  network  (in  appearance  similar  to  the  existing  social  network) 
that  depicts  the  possible  links  between  agents  on  the  basis  of  what  communications 
technology  they  carry  and  the  tagging  of  those  links  with  some  indication  of,  say, 
bandwidth,  latency,  spectrum  and  communications  protocol.  The  social  network 
produced  from  observations  could  then  be  compared  with  this  communications 
network  to  give  a  measure  of  network  usage  (this  would  have  the  useful  side  effect  of 
also  suggesting  to  the  analyst  alternative  routes  communications  might  take  that  are 
not  being  exploited).  Communications  between  agents  are  already  tagged  with  their 
mode  of  transmission  (e.g.,  verbal,  electrical  and  mechanical)  in  the  main  data-table 
and  through  the  CASA  data  collection  tool  and  this  taxonomy  could  be  extended  to 
encompass  more  specific  types  of  information  transfer  technology.  With  this  in  place 
it  could  act  as  a  constraint  upon  some  scenario/operational  designs  and  allow  one  to 
examine  more  fully  the  nature  of  communications  flow  within  the  system  as  a  whole. 
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Human  factors  and  cognition  as  a  systemic 
aspect  of  NEC/NCW 

WESTT  software 

-  Theoretical  framework  and  the  tool  to  do  it 
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